-
Notifications
You must be signed in to change notification settings - Fork 452
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Improve documentation of using LLMs #1636
Conversation
Codecov ReportAll modified and coverable lines are covered by tests ✅
Additional details and impacted files@@ Coverage Diff @@
## main #1636 +/- ##
==========================================
+ Coverage 80.33% 80.35% +0.01%
==========================================
Files 95 116 +21
Lines 6602 8194 +1592
==========================================
+ Hits 5304 6584 +1280
- Misses 1298 1610 +312 ☔ View full report in Codecov by Sentry. |
90fd25b
to
117ab10
Compare
db20a8e
to
6db444d
Compare
def _generate_wrapper(self, prompt: str, **kwargs: Any) -> str: | ||
""" | ||
Wrapper for the _generate method to handle exceptions. | ||
""" | ||
try: | ||
return self._generate(prompt, **kwargs) | ||
except Exception as e: | ||
logging.error(f"Error generating response for prompt '{prompt}': {e}") | ||
return "" | ||
|
||
def _batch_generate(self, prompts: List[str], **kwargs: Any) -> List[str]: | ||
""" | ||
Base method to batch generate text from a list of prompts using multi-threading. | ||
Handles exceptions in _generate method. | ||
""" | ||
with concurrent.futures.ThreadPoolExecutor( | ||
max_workers=self.max_batch_size | ||
) as executor: | ||
results = list( | ||
executor.map( | ||
lambda prompt: self._generate_wrapper(prompt, **kwargs), prompts | ||
) | ||
) | ||
|
||
return results |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Use multi-threading instead of async function for concurrent API requests. Because some versions of Jupyter do not directly support asynchronous requests, and this is easier to maintain and understand.
- Support dialog format messages - Delete the model class in vllm openai format - Use multi-threading instead of async function for concurrent API requests
|
||
|
||
|
||
## Support Framework/API |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Supported?
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Just one small typo. Great work.
Description
Related Issues
Checklist
make unit-testing
andmake integration-testing
successfully?Additional Notes or Comments